Bnb/dev #173
Bnb/dev #173
Conversation
…d to make sure different sets of features are indexed carefully. cape added to era_downloader. added vortex mean based bias correction code.
…on used for topo.
…ata and apply repeat after loading cache.
…same features originally but this isnt required. means/stds have to be indexed carefully.
…same features originally but this isnt required. means/stds have to be indexed carefully.
…spatial fwp with netcdf output and then temporal fwp on that output)
bnb32
force-pushed
the
bnb/dev
branch
2 times, most recently
from
7cc968f to
91f2b04
Compare
…d to make sure different sets of features are indexed carefully. cape added to era_downloader. added vortex mean based bias correction code.
…spatial fwp with netcdf output and then temporal fwp on that output)
sup3r/bias/bias_correct_means.py
Outdated
| """ | ||
|
|
||
| def __init__(self, path_pattern, in_heights, out_heights, overwrite=False): | ||
| """Parameters |
There was a problem hiding this comment.
Minor but i wonder if you need a line break to get the parameters to show up in the docs page
| @@ -0,0 +1,949 @@ | |||
| """Classes to compute means from vortex and era data and compute bias | |||
| correction factors. | |||
There was a problem hiding this comment.
Do you want to include any details on where / how to download the vortex means? No code just a quick tip.
| """Parameters | ||
| ---------- | ||
| path_pattern : str | ||
| Pattern for input tif files. Needs to include {month} and {height} |
There was a problem hiding this comment.
surprised no assert statement in the init for these format keys
There was a problem hiding this comment.
Me too. What was past Brandon thinking?
| ): | ||
| """Read vortex tif files, convert these to monthly netcdf files for all | ||
| input heights, interpolate this data to requested output heights, mask | ||
| fill values, and write all data to h5 file. |
There was a problem hiding this comment.
Seems kind of weird that you need the netcdf intermediate step but im assuming you thought of this and need it
There was a problem hiding this comment.
Yeah this was just an easy way to combine all the heights (xr.open_mfdataset()) in a single file which could then go through the normal interpolation routine.
sup3r/bias/bias_correct_means.py
Outdated
| logger.info(f"Finished writing means for {out_file}.") | ||
|
|
||
| @classmethod | ||
| def run(cls, era_pattern, years, features, out_pattern): |
There was a problem hiding this comment.
The ERA netcdf files should be able to be handled by the standard DataHandlerNC classes and the BiasCalc class. Why didnt you just convert vortex data to a standard format and then use the BiasCalc class from there? I think the mods to allow the BiasCalc module to use monthly data would be really simple
There was a problem hiding this comment.
Good point. Classic example of working harder and not smarter.
| max_workers=self.hr_dh.norm_workers) | ||
|
|
||
| @property | ||
| def output_features(self): | ||
| """Get list of output features. e.g. those that are returned by a | ||
| GAN""" | ||
| return self.hr_dh.output_features | ||
| return self.lr_dh.output_features |
There was a problem hiding this comment.
This is surprising, maybe i'm missing something, why wouldnt the high-res data handler have the correct output feature list? Just because the low-res handler has a list of "train only" features that makes this correct?
There was a problem hiding this comment.
High-res should have the correct output feature list but it just felt cleaner to use lr_dh for both self.features and self.output_features.
| @@ -219,7 +222,7 @@ def _run_pair_checks(self, hr_handler, lr_handler): | |||
| assert hr_handler.val_split == 0 and lr_handler.val_split == 0, msg | |||
| msg = ('Handlers have incompatible number of features. ' | |||
| f'({hr_handler.features} vs {lr_handler.features})') | |||
| assert hr_handler.features == lr_handler.features, msg | |||
| assert hr_handler.features == self.output_features, msg | |||
There was a problem hiding this comment.
Oh maybe this confirms my previous question?
…nform to format expected by classes in bias_calc.py. Added scalar only class to bias_calc.py
|
@bnb32 my only questions is whether we want to keep the |
Bc code for using vortex vs era means to compute factors. Some small tweaks for dual handler use with different sets of features for lr and hr handlers.